Skip to content

gpu-config [10/13]: add the GPU configuration manager main loop - #3256

Open
henry118 wants to merge 10 commits into
NVIDIA:mainfrom
henry118:gpu-config/10-applier
Open

henry118 wants to merge 10 commits into
NVIDIA:mainfrom
henry118:gpu-config/10-applier

Conversation

@henry118

@henry118 henry118 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Part of #3246 (10/13). Depends on: #3247 to #3255. Review only the last commit.

Adds internal/gpuconfig/applier: the manager's per-node loop that binds to
a driver load, decides once, and publishes status. Not used yet.


Devin Review

@henry118
henry118 requested a review from a team as a code owner November 17, 2026 19:03

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

The changes add a GPU configuration runtime. It parses and matches configuration documents, checks driver and workload evidence, applies power, clock, and ECC settings through NVML, and records outcomes in a locked state file. Kubernetes clients read nodes, pods, DRA resources, and ConfigMaps. A gate and applier coordinate decisions for driver loads, while status publishing updates node labels for sync state, schema version, and pending ECC activation.

Merge Risk: Moderate · up to f9ca8

This adds the GPU configuration manager's per-node loop and its supporting decision, parsing and gating logic. It is not yet wired into the operator, but several logic gaps should be fixed before it ships. A failed init container can permanently skip configuration for a driver load. Certain merged YAML entries can bypass validation and apply the wrong settings. The startup gate can release early after a brief module observation. Address these before merging.

Security architecture risk: Minimal · up to f9ca8

The inspected change establishes no material new security or failure-containment risk. GPU changes are guarded by workload and driver checks, and failed applications do not record success or release an expected startup gate. The new runtime is not activated by this change.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — When activated, the privileged setting path operates across the configured node’s enumerated GPUs, not merely one workload’s allocation. A shared document can therefore influence multiple consuming nodes. Current production reachability and deployment-granted authority are not established by this staged addition.

Trust Boundaries and Controls

  • observed — The configuration-to-GPU transition is guarded by allocator-release evidence, workload checks, driver readiness, load-ID revalidation after device handles open, and range checking before any setting write. Workload exemptions are scoped to the configured operator namespace rather than arbitrary tenant labels.

Resilience and Maintainability Implications

  • observed — Failure containment separates hardware application from administrative status publication. Application failures leave no successful decision for an expected gate to accept. Explicit gate bypasses are recorded as evidence that prevents a later unsafe application; failed status publication queues a retry without reapplying settings.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

Other comments: 2

internal/gpuconfig/gate/gate.go: 1

185-187: Minor · Functional Correctness · Quick win

Reset the absence timer when the locked recheck sees a module.

After NoModuleBound expires, the first LoadID call can report absence while this recheck reports presence. This branch leaves moduleAbsent true. If the module disappears before the next poll, the gate immediately records module zero instead of waiting for a new absence interval. A later applier then skips configuration.

Set g.moduleAbsent = false before returning here. This preserves the existing timeout for continuous absence and restarts it after an observed load.

internal/gpuconfig/decision/decide_test.go: 1

669-673: Minor · Functional Correctness · Quick win

Reject unexpected errors in successful test cases.

If a successful case returns Result{} and an error, this branch accepts None, unchanged state and closed NVML. It then returns before checking the expected outcome. For example, the settings-application case can pass without applying settings.

Require NoError when neither wantErrIs nor wantFatal expects an error. As per path instructions, flag assertions “that would still pass if the behavior under test were broken.”

Source: Path instructions

Nitpick comments: 3

internal/gpuconfig/apply/apply_test.go: 1

209-209: Trivial · Maintainability & Code Quality · Quick win

Add a range-failure case for the second GPU.

Every error case makes GPU-a fail. These tests could still pass if Apply validated only the first target.

Add valid GPU-a followed by GPU-b requesting 701 W. Require ErrRange and no writes on either GPU. This checks the all-target preflight contract.

As per path instructions, “Flag assertions that would still pass if the behavior under test were broken.”

Source: Path instructions

internal/gpuconfig/apply/nvmllib/nvmllib_test.go: 1

82-83: Trivial · Maintainability & Code Quality · Quick win

Check the device returned by DeviceByIndex.

These assertions would pass if DeviceByIndex(0) returned nil, nil. TestDevice does not catch that regression because it constructs device directly.

Give the mock device a known UUID. Retain the returned device and check its UUID through apply.Device. This detects a missing or incorrect handle wrapper.

As per path instructions, “Flag assertions that would still pass if the behavior under test were broken.”

Source: Path instructions

internal/gpuconfig/applier/applier_test.go: 1

457-463: Trivial · Maintainability & Code Quality · Quick win

Check the status produced by a changed document.

This test calls DocumentChanged but leaves staticDocs unchanged. It checks only node-read counts. It would pass if publishing continued to use an obsolete document snapshot.

Start with an applied document and InSync. Replace the document, send the notification, and assert OutOfSync after the jitter. Also assert that settings are not applied again. As per path instructions, flag tests where “the test only checks that a mock was called.”

Source: Path instructions

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 246a9ca and f9ca861.

Files ignored due to path filters: 39

  • go.sum is excluded by !**/*.sum, !go.sum
  • vendor/github.com/NVIDIA/go-nvml/LICENSE is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_linux.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_other.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/api.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers.h is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers_static.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const_static.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/device.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/doc.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/dynamicLibrary_mock.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/event_set.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/gpm.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/init.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/lib.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/memory_limits.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/README.md is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/computeinstance.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/device.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/eventset.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/extendedinterface.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpmsample.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpuinstance.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/interface.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/unit.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgpuinstance.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgputypeid.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.h is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/refcount.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/return.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/system.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/types_gen.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/unit.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/vgpu.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/zz_generated.api.go is excluded by !vendor/**, !**/zz_generated.*.go
  • vendor/modules.txt is excluded by !vendor/**
Files selected for processing: 36

  • go.mod
  • internal/gpuconfig/applier/applier.go
  • internal/gpuconfig/applier/applier_test.go
  • internal/gpuconfig/apply/apply.go
  • internal/gpuconfig/apply/apply_test.go
  • internal/gpuconfig/apply/fake/fake.go
  • internal/gpuconfig/apply/nvml.go
  • internal/gpuconfig/apply/nvmllib/nvmllib.go
  • internal/gpuconfig/apply/nvmllib/nvmllib_test.go
  • internal/gpuconfig/config/document.go
  • internal/gpuconfig/config/document_test.go
  • internal/gpuconfig/config/match.go
  • internal/gpuconfig/config/match_test.go
  • internal/gpuconfig/config/schema.go
  • internal/gpuconfig/decision/decide.go
  • internal/gpuconfig/decision/decide_test.go
  • internal/gpuconfig/decision/evidence.go
  • internal/gpuconfig/decision/helpers_test.go
  • internal/gpuconfig/decision/pods.go
  • internal/gpuconfig/gate/gate.go
  • internal/gpuconfig/gate/gate_test.go
  • internal/gpuconfig/kube/client.go
  • internal/gpuconfig/kube/client_test.go
  • internal/gpuconfig/kube/documents.go
  • internal/gpuconfig/kube/documents_test.go
  • internal/gpuconfig/loadid/driver.go
  • internal/gpuconfig/loadid/loadid.go
  • internal/gpuconfig/loadid/loadid_test.go
  • internal/gpuconfig/publish/publish.go
  • internal/gpuconfig/publish/publish_test.go
  • internal/gpuconfig/statefile/file.go
  • internal/gpuconfig/statefile/file_test.go
  • internal/gpuconfig/statefile/state.go
  • internal/gpuconfig/statefile/state_test.go
  • internal/gpuconfig/status/sync.go
  • internal/gpuconfig/status/sync_test.go

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 2 remain after this review.

// and hide an overlap. Elsewhere it rejects keys sigs.k8s.io/yaml can't
// stringify, such as null, which it reports unreadably.
func checkKeys(data []byte) error {
var doc yamlv2.MapSlice

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Functional Correctness · Quick win

Preserve merged keys during entry-name validation.

If gpus contains <<: {1: entryA, "1": entryB}, strict YAML decoding accepts the distinct key types. JSON conversion then collapses both names to "1". In yaml/v2 v2.4.4, MapSlice decoding discards merged items, so walkKeys cannot reject the integer entry name. The document can therefore apply one entry's settings instead of failing validation. The dependency implementations establish this path. (raw.githubusercontent.com)

Use a key-preserving representation that includes merged mappings. Add a regression case with these two names and different settings. Keep the existing valid merge case supported.

Comment thread internal/gpuconfig/decision/pods.go Outdated
return false
}
st := statuses(pod)
if s, ok := st[toolkitContainer]; ok && s.State.Terminated != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Functional Correctness · Quick win

Require successful termination before recording a gate pass.

If toolkit-validation or gpu-config terminates with ExitCode=1, these checks still report that the allocator wait or gate passed. Decide then records Skipped. Later decisions return Idle, even if the failed container recovers.

Require ExitCode == 0 for these termination checks. Keep the later-container checks, which can independently prove progress.

Also applies to: 162-162

@henry118 henry118 self-assigned this Nov 17, 2026
@henry118
henry118 force-pushed the gpu-config/10-applier branch from f9ca861 to 58be467 Compare November 18, 2026 23:11
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
@henry118
henry118 force-pushed the gpu-config/10-applier branch from 58be467 to f0e6101 Compare November 19, 2026 00:16
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
@henry118
henry118 force-pushed the gpu-config/10-applier branch from f0e6101 to b88cc63 Compare November 19, 2026 00:50
@henry118 henry118 changed the title gpu-config: add the GPU configuration manager main loop gpu-config [10/13]: add the GPU configuration manager main loop Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant